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ABSTRACT OF THE DISSERTATION 


Randomly Sampled-Data 
Control Systems 

by 

Kuoruey Han 

Professor A. V. Balakrishnan, Chair 

The purpose of this dissertation is to solve the LQR problem with random time 
sampling. Such a sampling scheme may arise from imperfect instrumentation as 
in the case of sampling jitter. It can also model the stochastic information ex- 
change among decentralized controller to name just a few\ The original continuous 
time control problem can be transformed into an equivalent discrete time stochastic 
control problem. However, the transformed discrete plant is a system with random 
parameters. Thus, the original control problem reduces to finding the optimal con- 
trol for systems with random parameters. 

However the control system, while discrete, is nonlinear and hence the optimal 
control law is difficult, to obtain. In particular, the ” separation principle or cer- 
tainty equivalence principle ’ 1 or '’variance neutrality condition does not hold in 
general. In fact, one finds only approximation techniques yielding suboptimal con- 
trols in the literature such as the self tuning control, dual control and OLFO (open 
loop feedback optimal) etc. 

In this dissertation^ practical suboptimal controller is proposed with the nice 
property of mean square stability. The proposed controller is suboptimal in the 
sense that the control structure is limited to be linear. Because of i. i. d. as- 
sumption, this does not seem unreasonable. Once the control structure is fixed ,the 


x 



stochastic discrete optimal control problem is transformed into an equivalent deter- 
ministic optimal control problem with dynamics described by the matrix difference 
equation. The A -horizon control problem is solved using the Lagrange s multiplier 
method. The infinite horizon control problem is formulated as a classical mini- 
mization problem. Assuming existence of solution to the minimization problem 
,the total system is shown to be mean square stable under certain observability 
conditions. Computer simulations are performed to illustrate these conditions. 
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Chapter 1 


Introduction 

1.1 Randomly sampled data control system 

The advent of computer era has made the use of digital signal processing prevalent 
in many research fields. In control field, we cannot avoid this trend and therefore 
the sampling of continuous time signal has become a very important process. I he 
optimal control theory for sampled data linear system has advanced rapidly since 
some of the early contribution. However, most of the systems considered in the past 
emphasized the use of uniform sampling scheme so that the information update in- 
terval is known deterministically. Of course uniform sampling scheme is inadequate 
in the diversified control field. For example, we cannot use that scheme to model the 
sampling jitter in the system. Nor can we use uniform sampling scheme to model 
the indeterministic information update process present in the distributed control 
system. In distributed control system, the local controller does not have a complete 
knowledge of the system description and have to rely on the information exchange 
among themselves in order to achieve satisfactory operation. Therefore, distributed 
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control system may have a stochastic information pattern where the data are ran- 
domly sampled. The above discussion lead us to consider the optimal control theory 
of systems with random time sampling. 


1.2 Orgonization of dissertation 

This dissertation is divided into six chapters. Chapter 1 is the introduction. In chap- 
ter 2, I formulate the continuous time control problem with stochasticly sampled 
data, where an equivalent discrete-time control problem with random parameters is 
derived. After reviewing some previous work in section 2, the fixed configuration 
approach is proposed to solve the control problem in section 3. The recursive least 
mean-square estimator is derived in chapter 3. Where it is shown that the esti- 
mator error covariance matrices depend on the particular control sequences. 1 hat 
is why the separation principle does not hold in this case. Certainly the certainty 
equivalence principle does not hold either. In chapter 4, necessary conditions of 
optimal control for N-horizon problem is derived. Although the control structure 
is fixed, the control is of closed loop type,i. e. ,it has the properties of dual control 
proposed by Feldbaum [15]. Next the formulation of oc-horizon control problem is 
proposed in order to obtain a linear time invariant system. It is shown that under 
some proper conditions the total system is mean-square stable assuming existence 
of optimal control. The infinite horizon optimal control problem for general one 
dimensional system is solved in chapter 5. An analytic expression for performance 
index as a function of gain is derived. The existence of optimal control and mean 
square stability of the total system are established for one dimensional system. The 
simulation results is presented ,in which the performance of the optimally controlled 
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system is compared to the system with the so-called certainty equivalent control. 
All in all the optimal control system is shown to perform better in the presence 
of sampling uncertainties. The summaries and conclusions of the dissertation are 
given in chapter 6. 



Chapter 2 

Formulation of control problem 


2.1 Stochastically sampled-data control system 

Consider the control problem shown in fig 1. The plant, which is a linear time- 
invariant system, is controlled by a computer. A digital sensor is used to collect 
data {v t } for controlling the plant. The computer analyzes the data {v*} and 
generates control signal {tq} which is then fed back through zero order hold device 
to the controlled plant. Here the sampling process {cq} is modeled by a stochastic 
process. This kind of stochastic sampling phenomenon is due to the imperfection 
in the sampling instrumentation. The stochastic sampling may even be used to 
model the time-sharing behavior among the centralized computer control system. 
The control signal is designed so that a quadratic cost functional is minimized. 

To be more specific, consider the following formulation of control problem. De- 
fine: 

Vi = v(<Ti) 
v l = • >Vi) 
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time shared computer 



Figure 1: Randomly sampled data control systems. 


b 





b+1 — <7 t +l — °l 


plant: 


x(t) = Ar(i) 4- Bu(t ) + FN (t) 

(1 

scT 

+ 

Cj 

II 

(2 


admissible control 

{«(•)!«(<) = Ui for t e [(Ti,(T,+ 1 ) and u t is measurable u 1 } 

performance index 

J = E r* {[x(t),Qx{t)] + [u(t),.Ru(<)]}^ (3) 

The above formulation is a stochastic continuous-time control problem. In 
the next section, the above control problem will be transformed into an equivalent 
discrete-time optimal control problem. In the sequel, we will call the above control 
problem SOCP. The controller has access of the measured data at discrete time 
Furthermore, the control u(t) is assumed to be constant within the sample interval 
and measurable v 1 ,i. e. it is nonanticipative . The state x is a n x 1 vector 
and the input u is a in x 1 vector. In general, the state noise N(t) is assumed to be 
a white Gaussian stochastic process with zero mean and unit spectral density. The 
inter-sample process {th} is assumed to be independent of the white noise process 
{N(t)}. Of particular interest is when tk is either a Macov process (especially a 
Marco v chain) or a i. i. d. process (independent identically distributed) which will 
be discussed in latter chapters. The control objective is represented by a quadratic 
cost functional. Notice that N in the cost functional 3 is fixed. The SCO P control 
problem may be stated as follows: 
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SC CP: Given system 1 and 2 with constrained control find a control sequence 
Ui,i = 0, • **,A r - 1 such that the cost functional 3 is minimal where u { is only a 
function of observations v l and past controls u 1 " 1 . 

2.2 Equivalent discrete-time control problem 

It is easy to show the problem formulated above can be transformed into an equiv- 
alent discrete-time stochastic control problem (see [12]). Let’s define 

Xi = x(a t ) 

m = v{a t ) 

We then obtain the following equivalent control problem: 

Discrete plant 

.r 1+ i = ^{t l+l )x t + F(t-+i)«i +6 (4) 

Vi = C-Xi + GOi ( 5 ) 


Performance index 

J 


N-l 


Y, E{[*(tk+i)xk,*k] + [/2(<fc+i)«fc,M fc ] 

k — 0 


+[W(t k+1 )uk,x k ] -(- [W(< fc+ i)*.n-,Ufc]} 


Admissible control u, is measurable v t . 
where 

(i = r e At FN(<r i+1 -t)dt 
Jo 


( 6 ) 


( 7 ) 
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m = 


(8) 

m = 

[* e AS Bd6 
Jo 

(9) 

*(h+ 1 ) = 

f tk+1 m'Qmdt 

Jo 

(10) 

R(h+ 0 = 

[* k+1 (T(tYQT(t) + R)dt 
Jo 

(11) 

mtk+i) = 

[ tk+1 $(t) m QT(t)dt 
Jo 

(12) 


The formulation above is a discrete time stochastic control problem with uncon- 
strained control. The plant is described by a random parameter difference equation. 
It is well-known that system (4,5) and criterion 6 can be transtromed to another 
equivalent problem without the cross term W(t k +i). In the sequel, we will assume 
W(tk+ 1 ) = 0 without loss of generality. Furthermore, it is usually assumed that 
the future sample interval is independent of past state and control {x k , Mfc}. 
we have the following result: 

E[9(tk + 1 )**,**] = E[E[^(t k ^)}x k ,x k } 

= E[T( 7^T)x k ,x k } = E[Q k x k , x k \ (13) 

Similarly, we have 

E[R(tk+i)u k <u k ] = E[R(tk+i)u k ,u k \ 

= E[H k u k ,u k } (14) 

where \&(£*. +1 ) = Q k and 1 ) — H k . Note, in the case {£k+i} process is i. i. d. (in- 

dependent identically distributed), Q k = Q and H k = H . In the sequel, we will 
also assume {A(/)},{0*} and {t t } are mutually independent process. If A(/) is a 
Gaussian process with mean zero ,then so is £* when conditioned on {<,■} process. 
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Actually, we have the following result: 

£[«*!*, +.] = e M FF-e A '‘dt ( 15 ) 

J 0 

Let's define 

F(t i+1 ) = [J* t+1 e At FF*e A y /2 ( 16 ) 

Threfore we can define 

& = F(ti + i)rji (17) 

where { //, } is a. Gaussian Process with mean zero and independent of { t , } process. 

From the above consideration, let’s summarize the formulation of control prob- 
lem as follows. In general, we are actually facing a control problem with random 
parameters both in dynamical system and peformance index. 

Plant 

.r,+ 1 = $(i,+i):r; + r(f t+ i)u,- + F(f, + i)r/; (lb) 

Vl = C.r, + GO, ( 19 ) 

where { ?/, }, { 1 , } and { 0, } are mutually independent random processes. 
{ 6; } and { ij, } are white Gaussian random processes with unity spectral 
desity. { t ; } is independent identically distributed ( i. i. d. ). 

Performance index 

.J = E Y'Uxk, Qxk) + K, Hu k ]} + E[x„, Qx n ] (20) 

0 

where Q > 0, H >0 and Q > 0. 

Admissible control u t measurable { }. 

control problem find { u,}q - 1 such that the performance index (20) is minimized. 
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2.3 Historical Perspective 

There has been a fair amount of work on randomly sampled data system. The 
stability of such system was studied by Kushner and Tobias (1969) [20], Agniel 
and Jury (1969,1971) [1] and Beutler (1972) [8]. The optimal control problem was 
studied among the others by Kalman (1961) [19], Montgomery and Lee (I960) [21] 
and De Koning [10. 14, 13, 12, 11]. 

Assuming perfect observation (i. e. C = I ,G = 0 ),no control cost ( H = 0 ) 
and i. i. d. sampling ( {/*} b b d. ), Kalman succeeded in finding the optimal 
control by applying dynamic programming principle: 

tit = -E[r m P(t + i)T]^E[rp(t + !)$]** (2i ) 

where P(t) satisfies nonlinear equation of Riccati type. He also solved the infinite- 
horizon problem in this case. 

Under the assumption that sample interval { t t } is a Marcov process, u t is 
measurable v l and U = (f,, • • * , U) and perfect observation, Montgomery was able 
to derive the optimal control by applying the dynamical programming theory. The 
optimal control is as follows : 

«: = #*.*,■ ( 22 ) 

Kilt, = — {E U [H + (23) 

where / J \ ~i~i satisfies Riccati type equation. The optimal control depends on 
past sample interval and past observation. It is preferred that the optimal control 
depends only on the past observation since sometimes { / ,} is not available. 

De Koning still assumes perfect observation ( ( ' = I,G = 0), extended Kalman s 
result to a general continuous time system and quadratic cost in the infinite horizon 
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case. Using the terminology of mean square stabilizability and detectability, he 
successfully showed the convergence of Riccati equation and existence of optimal 
control. He also proved the closed- loop system is mean square stable. His result 


can be summarized as follows: 

n* — —L B N-i~\qXi , i — 0, • * • , N — 1 (24) 

J*{x 0 ) = x* 0 B?Qx 0 Vx 0 (25) 

where Lx = (T*XT -f i/)^T*A<& (26) 

B*X = Wx$ - <FXr(FAT + + Q ( 27 ) 


Lx is the gain operator whereas B* is the associated Riccati operator. 

De Koning's result beautifully extends the familiar LQR theory to random pa- 
rameter system. However, the assumption of perfect observation is not available in 
practice. The purpose of this paper is thus trying to find a practical control under 
imperfect observation. 

The problem become extremely difficult under imperfect observation. The op- 
timal solution will be nonlinear and analytical solution is yet to be found. 1 he 
familiar separation principle does not hold in general. However under variance 
neutrality condition, the separation principle does hold. 

Definition: Variance Neutrality 

If E [(xi - Xi){x t - XiYlv^u^ 1 ] is independent of and id' 1 ,the system is 
called to have the property of variance neutrality. 

Assuming variance neutrality, it is easy to derive the optimal cont rol by applying 
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dynamic programming. See appendix A for the following formulas. 


( 28 ) 

(29) 

(30) 


Ui = - KiXi , 0 < i < N — 1 

A' = [T^qTp^~^W + //r'r *(g + /^_ t -_,)$ 

- A';r*(Q + /^_ i _ 1 )^ 

Even though the separation principle principle hold, x , = E(x t \v' ) is not linear in 
general and analytical solution is not possible. 


2.4 Fixed Configuration Approach 

With all the difficulty considered above, it is natural to obtain a linear optimal 
solution, i. e. restricting the control structure to be linear and trying to find an 
optimal control in linear class. Once limited to linear configuration as shown in 
fig. 2, the control problem is simplified into finding the optimal linear dynamic 
matrices which minimizes the perfermance index given in equation 20. 

Dynamics of Controller 

= E n X n (^) 

Xn = A n -\Xn-\ + B n - iV n (32) 

Control Problem min iv . r. t. {/v n ,/4 n , 

To further simplify the problem, we can assume that the controller is composed 
of a linear Kalman filter and a gain matrix as shown in fig. 3. This is equivalent to 
requiring: 

E[(x n - .r„K] = 0 for 1 < i < n (33) 
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Discrete Plant 


v t 

-m 


Optimal Linear Control 


Figure 2: proposed linear control structure 
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Figure 3: proposed linear separable control structure 
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Therefore x n is just the optimal linear mean-square estimate of state x n . It turns 
out that the Kalman filter is a function of { I\ n },i. e. ,A n and B n in equation 32 
depends on I\ n . furthermore, the error covariance matrix [R(x n — x n ){x n — ] is a 

function of I\ n also. This is why the separation principle does not hold in this case. 
However, in the usual SLQR problem, the error covariance matrix for the Kalman 
filter is independent of A and then the controller and estimator can be designed 
separately. 
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Chapter 3 


Optimal Linear Estimator 

In this chapter, the orthogonality principle is applied to derive the optimal linear 
least mean-square estimator of the following dynamic systems. 

.r, + 1 = ${t k+ i)x k + T(t k+1 )u k + F(t k+1 )r lk (34) 

v k = Cx k + CB k (35) 

let’s define: 

£’[.r t |e <: ] = optimal linear estimate of x k in terms of v k = {t’i, ■ ■ ■ ,v k ] 

,i k = £[x*|u A '] 

x k = E[x k |i’ fc_1 ] 

£(t;*) = linear subspace spanned by v k 

where optimality refers to minimization of the mean square error E[\\x — ^|| 2 ]* 
is the optimal linear one-step predictor of The Kalman filter is derived under 
the assumption that ujt = A ^Xk- It turns out that the error covariance matrices 
, hence the filter itself, is dependent on the particular control sequence {«*}• 
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3.1 Orthogonality Principle 

Let’s prove the following version of orthogonality principle. 

Theorem 1 Let x,y be n x 1 and m x 1 random vectors respectively. 4, .4* 
and A £ R nxm matrices. Then 

E[x\y] = A m y iff E[x - A»y, Ay] = 0, Vi € /? nxm (36) 

Proof: (sufficiency) If equation 36 holds, then we have 

\/A : E\\x - Ay \\ 2 = £||x - A.y + A.y - Ay\\ 2 
= E\\x - A,y\\ 2 + E\\(A. - A)y\\ 2 > E\\x - A.y\\ 2 
The minimum occurs when A = A*. 

(necessity) let's compute 

~E[x - (A. + \A)y,x - (T, + A A)y) ||a=o 

(l /\ 

= —2 E[Ay, x — A*y] = 0 
=► E[x - A.y, Ay] = 0, VT e iT* m □ 

3.2 Derivation of least mean-square estimator 

To derive the Kalman filter equations, first notice that the following identities hold: 

(0 i } k = £[ujfc|v fc_1 ] = Cxk ) 

proof: Obviously dk € £(i’ fc_1 ). 

Vi < k - 1 : E[(vk - Cx k )v*] 
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= E[(Cx k + Gr)k - Ci fc K] 

= E[C(x k - i,K + E[Gr, k v’\ = 0, □ 

(H) h = v k -v k 

= Cx k + Gr] k — Cx k (from equation 37) 

= Cx k + Gj] k (•&) 

(in) x k+ i = ^(/fc+i)^*: + r(ffc + i)«fc (-^9) 

where 6(t k+ i) = E[${t k+ ,)] and r(i fc+1 ) = E[T(t k+1 )]. Since {t k+ 1 } is assumed 
i. i. d. ,we will simply write = $ and T(£fc + i) = T hereafter. 

proof: Obviously, the RHS £ C(v k ). Furthermore, V? < k , we have: 1 

- ** k - r«*K} 

= £’{[$ fc .r fc + T k u k + F k r) k - $x k - ru fc ]r* } 

= E{{$ k x k - $x k ]v;} + E{[T k u k - fu fc K} 

+E{F k ri k v*} (40) 

The first term in equation 40 is equal to : 

E{<S> k ]E[x k v'} - E[*x k vt] 

= $E[x k v’} - $E[x k v*} = 0 

Since ifc+i is independent of x k and v*. Similarly, the second term is null. The third 
term is equal to : 

E[F k }E{ Vk v*} = 0 

1 In the sequel, we will simply write 

0(h+i) = $jfc,r(<* + i) = r*. and F(t k+ 1 ) = F k 


18 



Since Fktjk is independent of v“. □ 


(iv)xk = — Xk = one step predictor error (41 ) 

(v)xk = x k + A k v k (42) 

proof: We only need to prove that Xk — Xk — AkV k is orthogonal to {v k } foi 
certain appropriate A k matrices. For i < k — 1, we obtain: 

E[{x k - x k - /lfchfc)!’*] 

= E[(x k - Xk)v*} - E[AkV k v*} 

= 0, by orthogonality principle 
The following equation has also to be satisfied: 

E[(x k - Xk - A k vk)vl] = 0 

The RHS of equation 43 is equal to: 

A k E{(Cxk + Gi] k ){Cx k + GrtkY } (by equation 38) 

= A k {E(Cx k xlCn + GG*} 

= AkiCBkC* + GG a ) (44) 

provided we define matrix Bk as: 

B k = E[% k xl] = E[x k xl\ ( 45 ) 
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The above identity follows from: 


E[x k x* k ] = E[(x k - x k )(x k - x k + 

= E[(x k - x k )(x k - ar fc )*] + E[{x k - 

= E[{x k - x k ){x k - it)*] = £[***1-] ( 46 ) 

So, actually B k is the one-step predictor error covariance matrix. In the sequel, we 
will assume GG“ > 0 for convenience. In that case (CB k C* -\-GG*)~ l always exists. 
The left hand side of equation 43 is equal to: 

E[(x k - x k )(Cx k + Gi] k )*] 

= E[(x k - x k )x’ k C*] + 0, (by independence) 

= E[x k xl\C m 

= B k C * (4") 

by the same argument as equation 46. Combining equation 44 and 47, we obtain 
the following identity. 

(vi)A k = B k C~{CB k C m + GG*r l ( 48 ) 

The identity (v) is proved provided we can find the propagation equation for B 
The propagation equation will be found in the next section. Combining the above 
identities, we can derive the Kalman filter equation. 

x k +i = -G-+i + clfc+iG+i {by v ) 

= 4>x k + Tu k + A k+1 (v k+1 - V k +i) {by H and Hi) 

= $.i k + Tu k + A k+ i(v k+ i - Cx k + l) {by i) 

= <b.i' k + Tu k + A k+1 [v k+l - C{$x k + Tu k )\ {by Hi) (49) 
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Rearrange the above equation, we obtain the Kalman filter equation. 


{vii)£k+i = (I - A M C)$x k + (I - A k +iC)Tuk + A k + iu*+i (^0) 

The above equation is fairly similar to the usual Kalman filter equation. How- 
ever, one thing worth pointing out is that B k depends on the particular control 
({tifc}) sequence. The above filter equation holds true as long as { u k } is a linear 
function of { v k }. In chapter four, we will combine the estimation and control 
problem together where we are interested in the linear optimal control of the form 
{ u k = I\ k x k }. Therefore, in the following section we will derive B k based on this 
particular control sequence. 
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3.3 Propagation equation for covariance matri- 


ces 

In the derivation of B k , we will assume u k = h k x k d- e - u k is the product of gain 
matrix I\k and optimal linear state estimator x k ■ The following notation will be 
used: 


B k = E[x k xl] 

= one step predictor error covariance matrix 

R k = E[x k x* k ) 

= second moment matrix of state 

- 4 > 

f, = T k - r 

= <*>k + TkI<k 
= $ ~ ffcA’fc 

= + TK k — T k K k 

= t - r,/ u . 

Let’s begin with calculation of x k . By definition, we have: 

x k +i = x k+1 - x k +i 

= Q k x k + T k u k + F k t] k - $x k - Tn k 
= + F k i} k - $ k x k 

= V k x k + T ( I - A k C)x k — $ k A k G6 k + F k i) k 


( 51 ) 

( 52 ) 

( 53 ) 

( 54 ) 

( 55 ) 


( 56 ) 


( 57 ) 


( 58 ) 
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where it is easy to show that: 


x = x k -(I- A k C)x k + A k GO k (59) 

Therefore, from equation 58 we obtain: 

B k+ 1 = VR k V* + W[i - A^AjKii - A k CY$- 
+z o vB k {i - A k cy$ m + Tf^ 

+%A k GG m A%** (60) 

where 

ZoX = X + X* (61) 

For simplicity, we have omitted the t k+k dependence in equation 60 due to i. i. d. as- 
sumption. It is easy to show that : 

Xk + 1 — ^k x k + h ^K^A^GOk 

-r,/u(/ - AtC)x k + F k i ]k ( 62 ) 

R k+ 1 = WTgW + r K k A k GG m A k Kll' M 

-ZoWbJT^ A k CyK‘ k V + TF 

TKjl - A k C)B k (I - A k CYK;Y- (63) 

Equation 60 and 63 constitute the propagation equation of covariance matrices. It 
is obviously that the covariance matrices depend on the { I\ k } matrices. Since 
they depend on the past control, the variance neutrality condition does not hold 
in this case. The { B k } matrices measure the accuracy of the estimate, and this 
information is used in deriving the suboptimal control in next chapter. Let s define: 

w k = £[.r fc r fc ] (61) 
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From equation 42 43 and 44, we obtain: 


W k = E[x k xt} + A k E^ k vl}Al 

= E[x k xl ] - E[x k r k ] + A k E[v k vl)Al 
= R k - B k + A k (CB k Cr + GCA) A 


Now let’s compute the error covariance matrix P k defined as follows: 


P k = E[{x k - x k )(x k - x*)*] 

= E[x k xl]-E[x k xt] 

= R k -W k 


From equation 48 and 65 we obtain: 

U k = Pk — B k + B k C A k 
=> R k - W k = P k = B k - B k C* A* k 

=► p k = B k (I - A k cr = (/ - 

From equation 48, we have 


Ak(CB k C’ + GCA) = B k C~ 

=> (/ — A k C')B k C A = ,4jtGG” = -PfcC* 

=* A, = WfG'G'T 1 


Substituting eq. 70 inte eq. 68, we have: 


Pk = [I ~ PkCA{GCA)~'C}Bk 


=» f 7 (/ + G*(GG*r'Gfi,) = B k 


pt * 




Wk +1 =A k+l CR k+1 + J k+l VW k r (76) 

=> VVfc + j = A k+ iC(Pk+\ + WWi) + •4+i v Hl k ^ 

=► Jk+ iWfc+i =^+i^*+i + J *+iW ( 77 ) 

It is easy to prove the following identity: 

J w = I - A t+l C = (/ + B t+1 CT)-> (78) 

Therefore exists, and by equation 77 we obtain: 

Wk+ 1 = (/ - A, + 1 C)- 1 A, + 1 f 'T, + I + (79) 
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From equation 72, we can show that the first term in the above equation is equal to 
Bk + 1 — Pk+i- Therefore, we obtain: 

Wk+i = T1F,T* + B k+ , - Pk + 1 (80) 

Furthermore, we can show: 

x k + i - -yjt+i = $k(-rk - h) + 'P*** + FkVk (81 ) 

B k+1 = WP^T + 7T* Hr (82) 

Summary: We have obtained the following equations for relevant covariance matri- 
ces ( P fc , B k . H 'k ): 


a+ 1 

= (I + Bk+ 1 C a C)- 1 B k+1 

(83) 

Bk+ 1 

= <s>p k <i>* + FF" + mv k 4r 

(84) 

w k+i 

• m v k v* + Bm - p k+1 

(85) 


where we have assumed that GG“ = /. If not , replace C by ( \/GCP ) 1 C and r„ 
by (\/GG*) _1 v n . One thing worth noting is that B k +i — Pk+i > 0. This is because 
Bk + 1 is the one-step predictor error covariance matrix whereas Pk+i is the error 
covariance matrix [6]. 

3.4 Asymptotic behavior of Kalman filter 

In the following consideration, we will assume u*. = A .r* since we are interested in 
obtaining a linear time invariant estimator. We hope to establish conditions for 
which P x and il, converge as i — ► oc. To that end, let’s first give some definition 
(see [13]). 
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Consider the following system: 


X{+i — < b(ti+i)xi + r(C+i) w t (^6) 

If Ui — K £i , then we have 

**+i = < 87 ) 

where®(C+i) = $(<,-+i) + T(C+i)A (d8) 

Definition 1: System 87 is called m. s. stable (mean square) if \\xi\\ 2 0 for all 

%o- 

Definition 2: ($,r) is m. s. stabilizable if 3A such that system 87 is m. s. stable. 
In view of m. s. observability consider the following system: 

J-'i+i = ${U+i)*i 

vi = cx i m 

Definition 3: (4>,C) is called m. s. observable if ||r>i|| 2 = 0, V? =>• xq = 0. 

Let S n denote the linear space of real symmetric n x n matrices and define the 
transformation A$ : S n — * S n by: 

A*s = m&,ses n (oo) 

It is easy to prove the following lemma. 

Lemma 1: Aq is linear and monotoned- e. , 

S > 0 =► A^S > 0, V i (91) 
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Lemma 2: System 87 is m. s. stable iff 


p(Aq/ ) < 1 


Lemma 3: System 89 is m. s. observable iff 


p-i 

£ A^.C'C > 0, P = n(n + 1 )/2 (92) 

i — 0 

where p{A<p) is the spectral radius of 


3.5 Steady state Riccati equation 

If P k P and W k -* U'.then from equations ( 84, 85) we have the following 
SSRE: 


p 

= ( I + BC*C)-'B 

(93) 

B 

= $ P$* + 'HE'!'* + FF* 

(94) 

W 

= 'iuEvr + p - p 

(95) 


Let’s assume p{A<\/) < l,and therefore p( -4^ ) < 1. Equation 95 becomes: 


W = (I - A*)-' (B - P) 


(96) 


Substitute the above expression into equation 94, we obtain: 


B = (/-i4*)(/-i4*) _1 (i4* - A^Aty - A*) 

(/ - A^-'P + (J - A*)(I - Ait)~ l ~FF* (97) 
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Define: 

B = {I-A^-'B (98) 

P = {I-A^-'P (99) 

Then equation 93 and 97 become: 

P = {I + BC*C{I - Ay)]- l B (100) 

B = (I — /M -1 (/W - AtAq, - Aq)P + (I - A*)-'TT* (101) 

Note 1: A*-A*Ai-At = + is not necessarily monotonic. 

Note 2: If I\ = 0,then 'P = $. We have: 

P = [I + BC*C(I - A (102) 

B = A$P + (/- A^-'TF 

= $P$* + (I - ,4$) _1 FF* (103) 

In this case, it is shown by De Koning [9] that error covariance matrices Pk and li*. 
converge and there exist a steady state Kalman filter provided /?(/!$) < l,i. e. ,the 
system is mean square stable. 
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Chapter 4 

Derivation of linear optimal 
control 

4.1 Equivalent control problem 

The control problem formulated in Chapter 2 is solved in this chapter. The conti ol 
structure is limited to be a Kalman filter in series with the gain matrix as shown in 
fig. 3 of chapter 2. What remains to be done is the selection of the gain matrices { 
K n } since we have already derived the Kalman filter equation associated with the 
control structure. 

Let’s write down the equations for generating the optimal control { u n }: 

x k +i = {I — A k +iC)$x k + (/ — Ak+\C)Tiik + j4* + irfc+i (104) 

i*k = K k Xk 

Ak+\ - Pk+\C*(GG*)~ l - B k + x C*(CB k + x r + GG*r l 

where is the Kalman gain matrix. The control { u n } is selected such that the 
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performance index Jv is minimized. 


N 

J N = E^ixlQxk + ul^Hu^} (106) 

1 

= jr{trQR k + triq^HK^Wk.,} (107) 

1 

where from equation 65 of chapter 4, we know: 

W k — Rk — Bk + A k (CB k C * + GG*)A\ ( 108) 

Let’s define: 

[A,B] = trAB* (109) 

as the inner product in the space of n x n matrices. Equation 107 is then rewritten 

as: 

Jn = £{[<?, /fe] + [K k - iHK k -i, i ] } (110) 

1 

The relevant progation equation for covariance matrices {R k } and { B k } are from 
equation 60 and 63 of chapter 4: 

B k+l = TrJt + ¥(7 - A k C)B k {I - A k C'Y%* 

+2 O ¥bJj - A*C)*$* + TW 

+VA k GG*Ai$r (Hi) 

R k+l = WTW -1- TK k A k GG*AiW 

-z o Wbui - AkCyKtr* + T7* 

+TK k (I - A k C)B k (I - A k C)'K;r* (112) 

It is noticed that our stochastic control problem has been transformed into an equiv- 
alent deterministic control problem in which the state dynamics and performance 
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index J ]\ are given by equation 110 111 and 112 respectively. However, the equiv- 
alent control problem is nonlinear for Jn and state dynamics. Once the control 
structure is fixed, the sufficient statistics become { Rk,Bk The optimal contiol 
sequence { ?/* = I\\ } is selected based on the information about the sufficient 

statistics-the error covariance matrix of state estimator B \ t and the signal covariance 
matrix Rj<- 

There are many methods which can be used to solve this problem, such as dy- 
namic programming and minimum principle to name just a few. Furthermore, the 
technique in solving the nonlinear optimization problem can be used to obtain nu- 
merical solution at least. Because of the nonlinearness involved, it is difficult to 
obtain analytical solution. In applying the minimum principle to get the necessary 
condition for opt imal control, one difficulty remains. Notice that the state dynamics 
is described as a matrix difference equation in stead of a vector difference equation. 
The matrix minimum principle proposed by Athans (1967) (see [4]) is useful to 
derive the necessary conditions. In the following section the Lagrange s multiplier 
method is adopted to obtain the necessary condition which is basically the same as 
those for the matrix minimum principle. 


4.2 N-horizon control problem 

In this section the Lagrange’s multiplier technique is used to derive the necessaiv 

condition for t he optimal control of the N-horizon control problem formulated in 

the last section. Consider the auxiliary cost j ^ defined as follows: 

AT-l 

Jw = j n + L W- Bk+i - vRr * - vJkBkW 

0 

-2 O VBkJ;' & - 4fAkGG*A%4r - FF*} 
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+ LU, Rk + 1 - - TK k J k B k j;K;V 

0 

+2 o - rA'*yUGGMJA’**l’* - FF*} 

+ £{4, A* - B k C*{CB k C* + GG'*)- 1 } (113) 

1 

where { a ^ } , { ' /.- } and { 6* } are Lagrange constant matrices. We are free to set 

a* = <**,7* = 7* ( 114 ) 

because {F*,} and {R k ) are symmetric matrices. Our task is then boiled down to: 

min J,v u>. r. t. {[/u-lo^ -1 , } (113) 

After taking the gradient of J n with respect to the above matrices, we obtain the 
required necessary conditions for optimal control. The gradient of J\ ( A ) with 
respect to A at A, is defined as follows: 

[V a-.Iv (A', MA'] = lim J N (X, + ASX) 

V r n -Iv = Q + lN-1 = 0 

J,V = Gv = 0 

V B n Jn — a N-l — 0 

From the above equations, we can obtain the following boundary conditions: 

lN-l = —Q 
Sn = o 

a N = 0 
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Taking the gradient matrix of . with respect to 72,, we obtain the following: 


V 1 < i < N — 1 : 

Q + K'Hh'i - *?«,■*,■ 

+7t-i - ^7^ = 0 

=► 7i-i = W + -Q- K*HKi 

Taking the gradient matrix of J/v with respect to Bi , we obtain: 


V 1 < i < N — 1 : Vj 3 ; Jjv = 

-K-HKi + C* A*I<* HKiAiC + 

-(/-A 1 C)4; Q| i 1 (/- A,C) 

-1^7 - A,n - (/ - 

-(/ - t 1 C)*A7TT^T7/v,(/ - a,C) 

+*jOTt\-(/ - A.-C) + (/ - AiCyK-rJWr 
-biiCBiC” + G'Cr)- 1 ^ 

+C*(CBiC m + GG* ) ~ 1 CB t 6 i ( C B t C* + GG*)~ l C 


Note: In the above derivation, we use the following identities. 

1. tr AB * = tr B m A =► [A, B] = [B*,A*] 

2. If A B both are real, then tr AB * = tr BA* = tr B* A. 

=> [A, B] = [B*,A*] = [B, A] = [A*, BT) 


3. — A'*f *7,^, = 4/"a,vi/, 


(116) 


(117) 


34 



Taking the gradient matrix of J\ with respect to .4 , , we have: 

VI < i < N - 1 : V Al J N 

= 2 A* H K i A i(C Bi C* + GG*) + 2^*0,^, (/ - A t C)B,C* 
+2^'o iViBiC - 2$T l a l $ t A t GG‘ 

+2I<:TfWiKi(I ~ A t C)B t C * - 2K:TJWiBiC- 
- 2 I\*TJiiTiKiAi GG* + 6i = 0 


= 2 [ — A* H Ki Ai{CB{C* + GG*) - - A l C)B,C* 

+K‘f*^¥iBiC* + fy^¥,A t GG* 

-K*TfWJ<i(I - AiOBiCT + K*TJW>B t C* 

+ A7 TJW*I<iA t GG*} 

= 2 a* (-// + + 170 , 1 ',) a, .4, ( c#c- + gg* ) 

+2 4>*a t 4>Ai(CBiC* + GG*) 

+2(A7PS* + A *r*o,4 - ^‘a,*)^* 

Taking the gradient matrix of J\ with respect to A,, we obtain: 


V0< /< A'-1:V Ai J iV 
= 27/ A, IT, - 2f*o,'I',A, 

+2f*o,4',(/- 4,C)/?,(/ - 4,6’)* - 2f*o,4,(/ - 4,675; 
+ 2 r*o,'k, 7 ?,(/ - ,4,67* + 2f*o,'M,6’6;*4* 


-217-4^/6 - 2 r* 7 ,r i A,(/ - A,C)B t (I - 4,6’)* 


- 2 T * 7 , T, KiAiGG* A* + 2 r* 7 ,r,A',(/ - A t C)B, 


+2V^ l B i (I-A l C)* = 0 


( 118 ) 


35 



HKiWi + f *cqf A',(7 - A t C)B t 

-Fa$ KiRi + f*Q,f - AiCY 

-f*a,fA',(/ - A t C)5,(7 - A.-C)* 

-f*aif A',-AiGG*4- - rSTAT.A, 

-FiTA',(/ - AiC)B t {I - AiCY 
-T^TKiAiGG*A* +F^TA',(/ - A,C)B t 
+T^YK l B l (I - AiCY 
= f*a,4>A, - - /1,-C’)* 

+^^$7?, - rS$5 t (/ - A,-cr 

= H Ki Wi - t'aitKi W t - T^TKi Wi (119) 


[H - f*«if - r^.TlA'iiZi 


= (Fa,-* + r*7i*)(fli - + B t C‘A*) 

= (f-a^ + Ft^) Hj (120) 


where we have use the following identities in deriving the above equation (see 65 
and 48 of chapter 3): 


Hq = R, - Bi + A^CB.C* + GG’)A] 

A, = BiC*(CBiC* + GGT 1 

=> V i : h'iWi = {H - f*o t f - F^rj-^Foi# + T m -fi<j>)Wi (121 ) 

where we have assumed that ( H — f*a,T, — r*7,T,-) exists. If not, the Moore Pen- 
rose pseudo inverse can be used. It can be shown later, that ( H — r*a,T,- — r* 7 ,Ti) 
is self adjoint and strictly positive definite provided H is. Rightnow,we will show 
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that K, can be defined an anything on the null space of H, . Let s give the following 
lemma. 

Lemma :For the N-horizon control problem, A; need only be defined on the 
range space of U ’,( denoted by 7 Z ( H , ) . On the null space of Hj ( denoted by A"( If , ) ) , A , 
can be assumed anything. 

proof: We know that 

Ui = A'.-x,-, W { = E[xiX*] 

X e N{Wi) =► Ex*. x,x‘x = 0 
=> x*x = 0 w. p. 1 

In particular, if x, £ A"( li,),we obtain 

x*ii = 0 =>• ii = 0 w. p. 1 

and therefore u, = 0 with probability one and A, can be assumed anything. In 
general, we know: 

Xi — P Xi "f Qxi 

where P is the projection operator associated with IZ(Wi ), while Q is the projection 
operator associated with A ( W , ) . 

Ui = Kiii = I\{Pxi + KiQii 

Since Q.i'i £ A (Il ,).we obtain: 

[i,-, Q.r,] = 0 w. p. 1 
[Qii, Qx t ] = 0 
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Qii = 0 w. p. 1 

=>• u, = KiPii 


Again in this case. A, can assume anything on the null space of Mq. □ 

In the sequel, we will choose A, equal to: 

A', = (// - r>J” - + r-7.^.) d 22 ) 

on both the range and null space of Wi. Using the above expression , equation 116 
can be simplified into: 

7,_i = $*7,$i + + q 

+Z O +TfW>) 

-K*(H -rS-W. U23) 

In the similar fashion, equation 1 1 7 can be simplified into: 

-o,_, = — A * ( H - f^f, - TfWi)K t 

+c*a*k:(h - r >J~ - \JW,)I<>AC 

-(/ - A t C)*$*a,4> t (/ - A,C) 

Zo(I- AiCrKtit^i + TfWi) 

-6i{CBiC* + GG m )~ l C 

+C*(CBiC * + + GG m )~ l C (124) 

Equation 118 can lie simplified into: 

—6,-/2 = —I\*(H - f*a,f t - 

+A7( + TTt^)^c* 

= 0 using equation 122 (125) 
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Let’s define: 


Zi = (H- f-Q.f, - r*7.T«) 


Equation 124 becomes: 

-o t _, = —I\*Z,K t + C'ATK'tZiKiAiC 
-(I- AiCyfya&iU - A,C) 

+zo (i- AiCy k* ( f * 

= (I - A t CyK*ZJ< t (I - A t C) 

-(/ - A,C)*$*a t 4>,(I - AiC) ( 126) 

Using equation 122, equation 123 becomes: 

7,-i = + $*a,$,-Q 

2 o A7(r*M7 + TfWt) - i<;z t K t 

= $*7,$, + <f)*Q t d> ! + 2I<* Z l I<i - A * ZiKi - Q 
= + I<:Z,K t -Q (127) 

Remark: If — o, and —7 i are assumed to be self-adjoint and positive semidef- 
inite,then from equation 126 and 116 we know that — o,-_ 1 and —7,-1 are again 
self-adjoint and positive semidefinite. Since the boundary conditions derived before 
are: 

ajsj — 0 and — 8^ — Q 

Therefore we know that —a, and — are self-adjoint and positive semidefinite for 
all i. From the definition of Z x ,we know it is self-adjoint and positive semidefinite. 
Therefore, the inverse of (// — f *a t T; — r*7 t T,) exists provided H is self-adjoint and 
strictly positive definite. In the sequel, we will assume H is positive definite. 
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Since I\ t can assume anything on the null space of W t ,the solution to the optimal 
iV-horizon control problem stated above is not unique in general. The optimal gain 
matrix A t given by equation 122 is the most obvious and simple choice. We will 
call it the regular solution. As seen in equation 122 ,the optimal gain matrix has 
similar structure as those mentioned in section 3 of chapter2. If we assume that /,■ 
is deterministic, then equation 122 becomes: 

Ki = [ff- 

because r = 0 and 4> = 0. If t t is a random process, then some correction must be 
made. 

From the above discussion, the following theorem is given: 

Theorem: Assuming existence, the necessary conditions for the optimal con- 
trol for the A -horizon control problem stated before is that there exist auxiliary 
covariance mat rices — o,: and —S t such that the following equations and boundary 
conditions hold. 


i<i = [h - r*ai? - + r* 7l $j ( 12 s) 

Zi = [H - f *Qit - FtJ] > o (129) 

-a,., = (/ - AiCyi\*ZiK t (I - AiC) 

-(I - AtCY^aMI - A % C) > 0 (130) 

-7 - $*0,$ + Q 

-K*{H - T*at - T^yT)Ki > 0 (131) 

Si = 0, a/v-i = 0,7 at-i = —Q (132) 
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Let's define: 



Pl b = PI = - 7.-1 

(133) 

then we have the following necessary conditions: 


p! = 

J{K* ZjKiJi + J’&P^QJ, 

(134) 

pr 

r i 

$*P[+ 1$ + $ m Pi+\& + Q - Z iki 

(135) 


P b N = o,P r N = Q 

(136) 

where J, = / — A t C and 



Ai 

= BiC m (CBiC* + GG*)- 1 

(137) 

z t 

= [H + t*P? +l t + T*Pr +i V] 

(138) 

Ki 

= -Z-'iT'P^+t'P^) 

(139) 


The above equations and equation 111 and 112 are the usual two point boundary 
value problem (TPBVP) which can be solved numerically using techniques such as 
shooting method and relaxation method etc. Let's now consider a special example 
where the sampling process is deterministic. In which case it is well known the 
optimal solution exist. 

Special case: cr(tk) = 0. 

When the sampling process is deterministic, $ and T are deterministic and 
therefore $ and V — 0. By applying the necessary condition derived before, we 
obtain: 

Ki = -(H + rp^ l r)- l (rp[^) 

where 

P[ = <t>*P[+ x <t> + Q 
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Not surprisingly,the above equation are consistent with the known result. Further- 
more it is not only necessary but also the sufficient condition for optimal control. 
The control is optimal in the nonlinear structure as well. 

The calculation involved in the N-horizon problem is extremely huge. Further- 
more, the total system derived is a time- varying one. Since our particular interest is 
in obtaining a stationary total system, we will consider the infinite horizon problem 
in the next section. 
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4.3 Infinite horizon control problem 

To formulate the infinite horizon control problem, the performance index has to be 
changed as follows: 

.7= lim -^E{Y^(x* k Qx k + u k Hu k )} (140) 

i\ 7 — ►oo ^ 

The control structure is assumed to he a linear time- in variant one, with one moie 
restriction that the total system is m. s. stable. Assuming m. s. stability of the total 
system, all the second order covariance matrices converge and so the steady-state 
Kalman filter exists. Therefore, we restrict the control structure to be a steadi- 
state Kalman filter in series with a gain matrix. The dynamics of the compensatoi 
is then written as follows: 

•rj+i = *£. + Tm* + A[» i+ i - C+it - CTii»] (141) 

u t = Kh 0 «) 

where A — PC*(GG*)~ 1 (143) 

Since we assume the total system is m. s. stable, we have: 

Pk - P, B k -> B 
W k -> lb, A k A 

and P, B and W satisfies the SSRE (see equations 93 94 and 95 of chapter 3): 


p = 

( I + BC*C)~ l B 

(144) 

B = 

+ FF* 

(145) 

W = 

^IT#* + B- P 

(146) 

R = 

P + W 

(147) 
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The performance index is equal to: 

J = lim — E{y^j[Q, Rk\ + [K*Hk , Wk\)} 

N-+00 yv j 

= [Q,R} + [K*HK,W] (148) 

The control problem is then minimize J with respect to [R, A , 14 ] under con- 
straint equations 144 to 147. So the stochastic control problem has been tians- 
formed to a simple minimization problem. It is obviously that the optimal gain 
matrix A exists provided {b} process is deterministic. In that case, from the LQG 
theory the total system is stable. In the present ca,se,{b} is assumed to be i. i. d. 
If the solution exist, we expect that the total system will be m. s. stable. Since 
equations 144 to 147 is actually a generalized Liyapunov equation , the total system 
will be m. s. stable if some form of m. s. observability conditions hold. 

In Koning's paper [13], the following lemma regarding the generalized Liyapunov 
equation is given. 

Lemma 1: Consider the transformation A : S n — > S n defined by: 

AX — A*XA,A random (149) 

and the equation 

X = AX + B, B random, B > 0 (150) 

then 3 solution A > 0 ,(A, B 1//2 ) m. s. observable => A stable, A > 0. 

Note: If B > 0 the above result will hold. 

Using the above lemma, we want to prove that the total system will be m. s. sta- 
ble provided there is a solution to the above control problem and F F > 0 and 
C*C > 0 — the simplest case of m. s. observability conditions. It is easy to show 
that the SSRE is just a version of generalized LiyapunoGs equation. Let’s define: 

e*+i = Xk+\ - £*+i 0 51 ) 
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Then using equat ion 73 of chapter 3, we can show: 


x k + 1 
Cjk+l 


= Ar 


x k 

e-k 


+ F X 


where(A'fc = I\ by assumption): 


vl h -T k K 

J k+ -J k+l Y k K + J k+l V 

FkVk 

J k +\Fki)k — A k +\GO k +\ 


Let’s define: 


Xk = 


R k Pk 

P k P k 


then we have: 


A k + 1 = A xe X k A* e + F re F* e 


Letting k — + oc,we get the SSRE: 


X = A Te XA* e + F xe F' e 


where 


F F* 

1 Xt l Xt 


FF* 


FF*J* 


JFF* JFF'J' + AGG* A* 
A = PC*(GG*)-\J = I - AC 

First, let's give the following two lemma: 

Lemma 2: If R > 0, P > 0 and R — P > 0,then A = 


R P 
P P 


(152) 


(153) 


(154) 


(155) 


(156) 


(157) 

(158) 

(159) 


> 0 . 
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proof: 


V.r, y : 


-j 

* 

- 

X 

X 

X 

m y _ 


m y _ 


= x*Rx + y*Py + x*Py + y*Px 


= x’(R — P)x + x‘Px + y*Px + x*Py + y*Py 

= x *(R - P)x + (x + y)*P{x + y)> 0 □ 

Lemma 3: If (PC > 0 and F F* > 0 =£■ F xe F* e > 0. 
proof: Obviously F x ?F* e > 0. Furthermore; 



- 

★ 

f .. 1 

V.r. y : 

X 

_ y _ 

F F * 

1 re 1 re 

, 

H >5 

i 


x‘FF*x + x*FF-J*y 


+y*JFF*x + y* AGG* A* y + y*JFF*J*y 

(x + ryyTF{x + r y ) + y *AGcrA*y = o 

=► A'y = 0 and (x + J*y) = 0 

> (0£C* + GG*)~ l CBy = 0, using eq. 3-48 

=» CBy = 0 or C*CBy = 0 

> By = 0 =*> y = 0, since i? > 0 using eq. 145 

=> .r = 0, using eq. 160 


(160) 


jF„F* > 0 □ 

1 xe A je ^ 


Now, we are able to prove the following theorem. 

Theorem 1: If the control problem has a solution, i. e. ,SSRE has solution 
(i? > 0,P > 0 and /? — P > 0) and 3 I\ minimizes eq. 148, then the total system 
is m. s. stable (i. e. Aj. e is m. s. stable) provided C*G > 0 and F F* > 0. 
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proof: If 3 solution (P>0,P>0andP-P>0)of SSRE, then X > 0 by 
lemma 2. Furthermore, F re F* f > 0 provided C*C > 0 and Ft* > 0 by lemma 3. 
Therefore, A Xf is in. s. stable by lemma 1 □. 

Note:The assumption FF 7 > 0 is not unrealistic. If {A,F) is completely con- 
trollable and l 2 > 0 with probability one, then we have: 

TF 7 = E { jf ,+ ‘ e M FF*e AU dt J > 0 

However, the assumption CPC > 0 is too restricted unless in one dimensional case. 
We hope it can be relaxed in the future. It is suspected that m. s. observability of 
(<J>,(7) and (4>*, F m ) will be enough. 

In the deterministic case, the control problem has a solution which is equivalent 
to the LQG control as the following example shows. 

Special case: a (Ik) — 0 

For simplicity reason, consider the one dimensional case. We are facing the 
following minimization problem: 

B = <t> 2 P + F 2 

P=( 1 +bc 2 )- x b 
yv = m 3 W + B - P,® = $ + TI\ 

J. X , = QP + (Q + HK 2 )W 
= QP + (Q + HK 2 )[l - ^ 2 r x (B - P) 

ini n J lX> w. r. t. (P, Wand A ) (161) 
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Obviously, B and B are independent of A . That is why the separation principle 
holds in this case. Setting 4^ = 0,we have: 

[HK + (Q + HK 2 )(\ - V 2 )- l *T](l - ^ 2 ) _1 (B -P) = 0 

=> -HYQK 2 + {H - H$ + QT 2 )I< + Q$T = 0 (162) 

Let's define: 

I< = -(T 2 P C + Hy'TPc* (163) 

Substituting the above expression into equation 161, we have: 

p c = <s> 2 p c - <t> 2 r 2 p?(r 2 p c + h)~ x + q (164) 

which is exactly the associated SSRE for the LQG problem as expected. 
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Assuming existence of optimal control, i.e., assuming that the minimization prob- 
lem formulated before has a solution, then we can derive a set of necessary conditions 
similar to those derived for the N-horizon optimal control problem. Applying the 
Lagrange’s multiplier method as in section 4. 2, the following necessary conditions is 

given. 

Theorem 2: Assuming existence, the necessary conditions for the optimal con- 
trol for the infinite-horizon control problem stated before is that there exist auxiliary 
covariance matrices — o and —6 such that the following equations hold. 


K = [H - f*af - (165) 

Z = [. H - f *af - PtT] H 66 ) 

- O = (7 - ACYICZK{I - AC) - (/ - AC)*Q*aQ(I - AC) (167) 

-7 = - 4>*a4> + Q 

—K*(H - PoT - T*~fT)K (168) 


The above equations in terms of —a and —7 are one variations of the fa- 
miliar steady state Riccati type equations. Which together with equations 141 
through 147 are essential to the solution of optimal control problem. Unfortu- 
nately, they are highly nonlinear as well as coupled set of equations. It is noticed 
that they are coupled through { A } and { A } matrices, the optimal control gain and 
optimal estimator gain respectively. All the difficulty in analyzing and solving the 
underlying optimal control problem comes from the coupling phenomenon, which in 
the case of deterministic uniform sampling problem the famous separation principle 
(instead of coupled effect ) can be used to simplify the optimal control problem into 
separate control and estimation problems. 
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From the above discussion, it is obvious that one way to obtain the optimal gain 
matrices { A } and { A } is trying to find the solutions of the above mentioned 
nonlinear set of coupled Riccati type equations (144 through 147 ,165 through 
168). There are many methods in the literature to solve the nonlinear systems of 
equation, e.g., the secant method ,the Newton- Ralphson method etc. 

The alternate way is to solve the minimization problem directly. This waj is 
preferrable because there are many efficient general techniques for finding the min- 
imum of a function of many variables, e.g., the simplex method, the steepest descent 
method, quadratic programming method, reduced gradient method to name just a 
few. 

In the case where the sampling scheme is deterministic ,we know that both 
and are stable. It is interesting to know whether 'P and J $ are mean square 
stable. The result is given in the following theorem. 

Theorem 3: If the optimal control problem has a solution, i.e., SSRE has a 
solution (/? > (U> > 0 M - P> 0) and 3 K minimizing equation 148, then $ and 
J<£ are mean square stable provided Q > 0 and F F* > 0. 

proof: First notice J defined as follows is nonsingular. 

J = / -AC = I - PC*C - (1 + BC*C)~ l (169) 

From equation 144, we have: 

P = (i-AC)B(I-AC) m + (I-AC)BC*A m 
= {1 - AC)B(I - AC)* + PCFCP 

= JBJ* + PC*CP ( 17 °) 
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Therefore from equation 145, we obtain: 


p = jopo*j* + jwo*j* + jff*j* + pc*cp (Hi) 

The latter three terms are strictly positive definite provided JFF*J is. (which is 
true since J is nonsingular.) By lemma l,we know that JO is mean square stable. 
Since we assume that the minimization has a solution, by applying theorem 2 we 
know there exist —7 and —a covariance matrices (both of them positive semidef- 
inite) satisfying equations 165 through 168. After some simple computation, we 
have: 

- 7 = - 0 * 7 # - 0 *a 0 + Q + ICHK (172) 

Again the latter three terms are strictly positive definite provided Q is. By applying 
lemma 1 again ,we know that 0 is mean square stable. □ 

Note: It is easy to prove that mean square stability imply mean stability. There- 
fore we also prove that 0 and JO are mean stable. 

The generalized Lyapunov equation given in equation 157 is essential in the 
study of mean square stability of the total system. If ( A* e ,F* e ) (see equations 
153 and 154) is mean square observable, i.e., 

0 (173) 

0 

where A xe is defined as: 

V X € R 2nx2n : A xe X = A X€ XA*„ 

then A xf is mean square stable by lemma 1. However, under what conditions is {A Tf 
,F xe ) mean square stable ? Recall the dynamic equation for the one step predictor. 

* k + 1 = 
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= ty(x k + Av k ) 

= Vx k + tyA(v k - v k ) 

= $(/ - AC)x k + *Av k 

= yjx k + VAv k (174) 

x k+ i = J^.T^ + Av k+l O' 5 ) 

The above is the optimal linear filter. It is interesting to know whether 
or (J'P,/!) are completely controllable. The following lemma is given. 

Lemma 2: Assuming ('k) _1 exists then the following statements are equivalent. 

1. ('kJ.'I'A) is completely controllable. 

2. (Jty,A) is completely controllable. 

3. (4', A) is completely controllable. 

4. (4 , , l ih4) is completely controllable. 
proof:(l) -£=>■ (2) Notice the following identities. 

('k)- 1 ['kJ]4» = J'F , (^) -1 [4 , A] = A 

Therefore (fyJ$A) and (JT,A) are similar state space representation of the same 
system and hence t he proof is completed. 

(3) •<=>■ (4): Follow the same argument as before, we have 

(4') _l [4']'i' = 'P , (4') -1 [<i'A] = A 

and hence the proof is completed. 
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(1) ==£■ (4): Assume 


Vi > 1 : A*(Vrx = 0 

=> = 0 
(*A)*(#J)*® = A*V*J*V*x 
= ,4T(/-CT)ri 
= A*^* 2 x = 0 


In general, by induct ion we have: 


('PA)‘('I>J)* l a: = 0, Vi > 0 


x = 0 


is completely controllable. 


(4) => (1): Assume (4) is true. 


By induction.we have: 


Mi > 0 : Jr® = 0 

=> = 0 

=* A'^r^'x = o 
=> A m $*(I - C*A*)<i*x = 0 
=► A* ty m2 x = 0 

Mr > 1 : A*($rx = 0 

=4* x = 0 
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=> (tyJ^A) is completely controllable. □. 

Theorem 4: (/!*,. F* e ) is mean square observable if and only if (^,A) is com- 
pletely controllable. 

proof: (sufficiency) Assume 


[C y“]A l xe {F xe F* e ) 


x 


= 0 V i > 0 


where ,4 re (.Y) = A xt X A m Xf ,. 


i = 0, [x y]*F xe F; 


x 

y 


= o 


A’y - 0, and (x + J*y) = 0(bv equationl60) 

=► r V = {I-C*A*)y = y 


x + y = 0 


=» y = - x , A‘x — 0, J*x = x 


( 176 ) 


Next, consider the case i ’ — 1 : 


x - x]*[A xe {F xe F; e )] 


= 0 


= [* - .rTA rr F xe F~ 

The latter two terms are equal to: 


-at* — a *r*j* + 


x 

— X 


vh*j- - J*x 


T»*.r 

— 7\ *r*.r + K*Y*J*x - $! m J*x 


-'J/Tr 


(177) 
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by using equation 176. 


Therefore,equat ion 177 imply the following: 



* 

'tTx 


F f* 

1 xe 1 xe 



By the same argument as before, we can prove: 

A*^*x = 0, and = ^*x 


By induction, we can prove easily the following: 


4*^ = 0, Vi > 0 


x = 0, and y = 0 


Since (>P,A) is completely controllable by assumption. The proof i 
pleted. 


(necessity): Assume that (A* e ,/^ e ) is mean square observable. 

A*V* i x = Q, Vi > 0 


=> = (/ - C*A*)9 mi x = vT’x 


Following the same argument as before, we can prove: 


r- 

* 

- 

.r 


X 

— X 

J* F F* 

^xe 1 xe 



Vi > 0 


ty mi x 

* 

#**.T 





(178) 


therefore com- 
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X 


— 0, i.e., x — 0 


L ~ x J 

=> The proof is complete. □ 

Note: From the theorem lemma 1 and lemma 2 ,it is obvious that the total sys- 
tem will be mean square stable provided the optimal linear estimator is completely 
controllable. 

Corollary 1: Assume existence of optimal control u ^ = A F&, the total system is 
mean square stable provided the dynamics of the estimator (^J^A) is completely 
controllable. 

Note: Once the mimimization problem is solved ,the above corollary can be 
used to test whether the total system is mean square stable. However ,it is only a 
sufficient condition for m. s. stability. 

Corollary 2: ( 4A.4) is completely controllable if both C*C and F F* are positive 
definite. 

proof: We know P > 0 if > 0 from equations 144 through 147. Furthermore, A 
PC*C is full rank provided C*C is. □ 

The above is a trivial case when the linear estimator is completely control- 
lable, and hence the total system is mean square stable as is proved in theorem 1. 

In general, it is difficult to prove under what conditions the linear optimal estimator 
will be completely controllable even in the case where the sampling process is a 
deterministic uniform one. 

Recall in the case of conventional LQG problem, in order for the total system 
to be stable it is enough to require the estimator to be detectable . Therefore to 
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get better result, somehow we need to introduce similar notion such as mean squat t 
detectability . This notion is first introduced by De I\oning.(see [10] and [14]) 
Consider the following system: 


Zi+i = 


(179) 


Vi = CiXi 


(180) 


where { <!>, } { C\ } are sequence of independent random matrices with constant 


statistics. 


Definition: is called mean detectable if 

157 = 0, Vi > 0=>x"^0 

and mean sqitan detectable if 

PTF = ov ? =► - o 

It is easy to see that mean detectability means that unobservable (mean) modes aie 
mean stable, and that the modes which are not mean square observable are mean 
square stable. The following lemma are proved by De Koning. 

Lemma 3: (<t>,C) is mean square detectable if and only if 

x^A^Uxo = 0 V i => x* 0 MU)*o -> 0 

where A<& is defined as: 

A* : S'" -> S n : MX = 

Lemma 4: Consider t he transformation A : S n — » S n defined by 

AX = A*XA, A random 
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and the equation 


X = AX + B, B random ,B> 0 

Then there exists a solution X > 0,and (A,/? 1 ^ 2 ) mean square detectable =$■ A 
stable. (i.e., A mean square stable) 

Applying the above lemma, the total system will be mean square stable provided 
(A; e ,F; e ) is mean square observable. First, the following theorem is given. 
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Theorem 5: If there exist R > OP > 0 and R-P > 0 satisfying the SSRE 
(equations 144 through 147) ,then (A* e ,F* e ) is mean square detectable provided ^ 


is stable and F F" > 0. 

proof: define the linear monotonic transformation A as: 


AX = A xe X A* e 


X 



X 




A‘(F„F;,) 


= 0 V* 

_ y _ 



. y . 




X 

* 

oo 



X 

=> 


Y,A'{F xe i 

F* ) 

xe / 



. y . 

i=0 



_ y . 


r 

-i* r 


i r 

1 


= 0 


x 

y 


R P 
P P 


= 0 


x *(R - P)x + (x + y)*P{x A y ) = 0 
=► x*Wx + (x + y)*P(x + y) = 0 
=> Wx = 0, &ar + y = 0 


because P > 0 provided F F* > 0. Since 

W = VWV* + B-P 

We have 

xAVx = x'VWfrx + x*(B - P)x = 0 
IF'Tj = 0, and (B - P)x = 0 

Furthermore, we know: 


B-P = B-{I + BC*C)~'B 
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= B - (I - PC*C)B 
= PC'CB = BC'CP 


= BC*A* 


BCTA'x = 0 


C* A* x = 0 


J*x — (/ - C*A*)x = x 


In general, we have: 


x e A”( H 7 ) => tyT e A' , (I / F)and x € A r ( J* - I) 


Therefore, we have by induction: 


Vi : 'Thr € ftf{ IT) and 9 mi x <E A'(/ - J‘) 
i.e. V i : WV mi x = 0 and J*$* l x = 


Next, we want to prove: 


- 

* 

- 

X 

— X 

(AI) 

X 

— X 


0 as i 


oo 



— .T 


- 


_ 

- 


- 

X 



^*J* 


X 







— X 


-AT* 

—I\*T*J* + J* 


—X 



* 


,r 

^xe 

- 'PVT 

— .r 


— A'T*z + A'TVT - 'PVT 


(181) 
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In general, we have by induction: 


1 

bt 

* 

i©* 

★ 

vTx 

-V*x 


-^*.r 




(A'l) 


—X 


X 

— X 


= 2x*^ l ^ l x — > 0 


The above is true provided $ is stable. □ 


Theorem 6: If the optimal control problem has a solution, i.e., SSRE has solu- 
tion (R > 0, P > 0,and R - P > 0) and 3 K minimize equation 148, then the 
total system is mean square stable provided F F* > 0 and Q > 0. 


proof: From theorem 4, we know that is mean square stable and therefore 
is mean stable, (i.e..^ is stable) From theorem 5, we know that (A re ,F re ) is mean 
square detectable and therefore A xt is mean square stable by lemma 4. □ 

From the above analysis, we know that the total system is mean square stable 
without requiring C*C to be positive definite as long as Q and / F * are positive 
definite. This is not surprising because it is the case in the coventional LQG problem 
(or when the sampling process is deterministic). In those two cases, the total system 
is stable provided (4>,/ 7 ) and are completely controllable.(or more loosely 

($,F) is stabilizable and ($,Q) detectable) Moreover, the observability of ($,C) and 
stabilizability of (4>,T) are only in connection with the existence of solutions of two 
famous isolated Riccati equation, (see [6]) In our problem, there is reason to believe 
some consistency will appear as the above theorem shows. 
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Chapter 5 


Examples and simulation results: 


In this chapter, the simulation results for randomly sampled data control is pre- 
sented. Althouglnour techniques for solving the optimal control problem as de- 
scribed in chapter 5 does not limited to control of descrete plant obtained this way. 
It applies equally well to discrete plant with random parameter such as those en- 
countered in chemical processes. However, my particular interest is in randomly 
sampled data control system design. The simulation of one dimensional system is 
presented t horoughly in section 5-1. In one dimensional case, the optimal control ex- 
ists under proper mean square stabilizability and observability conditions of ($,T) 
and (<I>,c) respectively. Ferthermore, the total system is mean square stable (also 
stable w.p.l) provided F F* > 0 and q > 0. The simulation of one dimensional 
system turns out to be very successful. It validate all the results of previous chap- 
ters. The optimally controlled system perform better as compared to the system 
with certainty equivalent control. The main reason is that the optimal system can 
guarantee the mean square stability of the total system whereas the latter system 
can only achieve mean stability. 
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5.1 One Dimensional System Simulation 


In this section ,\vo are dealing with the optimal control problem for the general 
one dimensional system. The continuous time plant with randomly sampled data 
control is described as the following one dimensional differential equation. 

x(t) = a c x(t) + b c u(t) + f c n(t) (18-2) 

v k — cx(a k ) + g9 k (183) 

where all the variables are one dimensional scalars. u(£) is the continuous time 
white Gaussian state noise. { Ok } is the discrete time white Gaussian observation 
noise. { (7k } is the sampling time process which is modeled as a stochastic process 
with independent increment. The performance index is given as follows: 

J= lim E f {q c x(t) 2 + h c u(t) 2 }dt ( 1 84 ) 

(J N — (Tq Jc to 

where we assume the intersample process ti = a x — cr t - \ is strictly positive and 
therefore <t\- — ► oc i(T A — ► oc. Furthermore, we assume that in our simulat ion { /, 
} process is independent indentically distributed and t % is uniformly distributed in 
the interval [Ai,A 2 ],i.e., 


and therefore 


0 < Aj < ti < A 2 < oo, i = 0, 1, 


Puiti) = t € [Ai, A 2 ] 

l±2 — 




EKti-ti) 2 ] = 


21 (A 2 -A,) 2 


(185) 

( 186 ) 
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Let and u t denote respectively x((Ti) and r/(eq). We may transform system \b2 
and 183, given sampling process { t t }, to the equivalent discrete time system as 
described in chapter 2. 

Xk+l = ^k x k + Tfctit + F k 1] k ( 187 ) 

v k = cx k + g9 k ( l88 ) 

where { <I> fc }, { 1\. } and { F k t] k } are sequences of independent random variables. 
Moreover, and 1 , are independent of F k i) k ,j ^ and uncorrelated with Fji) r 
Let’s compute . 1\. and F k random variables and their associated statistics. 


$ k = e actk (189) 

T k = t* e acS b c d6 = — (e“ c< * - 1) (190) 

Jo a c 

F k = f k e 2act f 2 dt = Il(e 2a ^ - 1 ) (191) 

J o c 

Assume is uniformly distributed in [Aj,A 2 ], we can compute their first and 
second order moments as follows: 

4 > 


r&2 

E[e actk ] = / 

J A i 




A, 


-dt 


= f fI(a c ;A 2 ,A 1 ) 


, (192) 

(A 2 - A i)a c 

In the sequel, we will simply write II(a c ) for simplicity, hollowing the same procedure 
as above, we obtain: 


r = — [n(« c ) - 1] 


$ 2 = Yl{2a c ) 


bl 


P = -|[n(2a c )-2n(a c ) + l] 


(193) 

(194) 

(195) 
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( 196 ) 


$r = — [n(2 a c ) - n(a c )] 

a c 


$2 = _ $)2 = $2 _ (|>)2 

(197) 

¥F = W - 4>r 

(198) 

F = n - (f ) 2 

(199) 

f 2 = 2L[n(*2« c )-i] 

Za c 

(200) 

The performance index 184 can also be transformed into an 

equivalent long term 

average sum criterion. 


hq = T 7 E[qx 2 k + hu\ + 2 WX k U k ] 

iy> k-o 

(201) 


In order to simplify the computation ,we will omit the cross term 2wx k u k hy setting 
w = 0. This does not lose any generality since our techniques apply equally well in 
the case w ^ 0. In the sequel, we will use the following performance index which is 
defined directly in terms of discrete time system variables. 

J = E[qxl + hul} ( 202 ) 

k= 0 

where it is assumed that q > 0 and h > 0. In the following ,we will try to solve 
the infinite horizon discrete time optimal control problem as described in chapter 
4. That is: 

Given system equations 187 and 188, find an admissible control that will 
minimize the performance index 202. 

Admissible means that u t is a function of v l = (iq, • • • , V{) and £’[||.r l 2 ||] converge as 
i — ► oo to the same constant value for all .ro. In essence, we try to seek an stationary 
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optimal control that will make the total system mean square stable. As in chapter 
4, in order to get an practical control, we will limit the control structure to be linear 
and separable. To be exactly, our controller consists of an optimal linear estimator 
in series with a gain operator. The dynamics of the controller is as follows:(see 
chapter 4) 

.i'k+i = + f 'u k + a{v k + 1 - - ctu k ) (203) 

u k = kh (204) 

where a = pc/ g 2 . Once the control structure is limited ,the infinite horizon optimal 
control problem can be transformed into a simple classical minimization problem. 


J := qr -\- k 2 hw 


= q(p + w) + k 2 hw 

(205) 

p = (1 + bc 2 g~ 2 )~ l b 

(206) 

b = <P 2 p + 'J 2 w + F 2 

(207) 

to = + b — p 

(208) 

vp = $ + TA 

(209) 

vp = $ + f A 

(210) 

ip 2 = F + 2$TA + Fa 2 

(211) 

vp 2 = W + 2Wk + PA 2 

(212) 


Our nonlinear optimization problem with equality constraint is then as follows: 

min J(6, p, ie, k) = q(p + te) + hk 2 w 
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subject to constraint equations 206 and 208. 

By applying theorem 4 of chapter 4 ,we know it is necessary that ^ is mean 
square stable if there is a solution to the above minimization problem. Since ^ = 
<I> -f rfr,this place additional constraint on k. Moreover, for the above minimization 
problem 5.1 to ha ve a applicable solut ion, we must require S to be nonempty. W here 
S is defined as: 

5= {ke R l : W+TW < 1} 

Basically, it is equivalent to require that ($,1") be mean square stabilizable. Let s 
give the following fact concerning mean square stabilizability of ($,T). 

fact 1: Assume T 2 ^ 0, then we have: 

(i) (*,n is mean square stabilizable if and only if 

^2f2 _ f2 _ ($f ) 2 < 0 


(ii) If i s mean square stabilizable then 

W ^/($T) 2 - (P - i)p W y^W) 2 - (jF - ur 2 
5 = ff ' + p 


proof: (4> ,T) being mean square stable: 


3 k such that ($ + Tk) 2 < 1 


=» <& 2 + + r 2 k 2 < 1 

2 $r, ^-1 n 

=> + -fr k ' + -Yr~ < 0 

$r \ 2 W-i /W \ 2 
, r 2 / r 2 l r 2 / 


(214) 


67 



Therefore, we have to require: 



==► ($ 2 -i)r 2 -($r) 2 <o 
=> WT 1 - r 2 - (W ) 2 < o 


Solving equation 2 1 4 ,we obtain: 

W v/($r) 2 -(^-i)P 

p2 fl 


<k< ~ff + 


y/(^r) 2 -^ 2 - i)r 2 n 
r 2 


Let’s define ki and k'h as follows: 

k ‘ — ~ f2 p2 

If ^(^T) 2 -(^-i)n 

h — pj p2 


(215) 

(216) 


fact 2: For one dimensional sampled data control system where the sampling 
process {t t } is uniformly distributed , we have: 


(i) ($/., I \) is mean square stabilizable. (where <t>k and T k are defined in equa- 
tions 189 through 191 ) 


(ii) S = is nonempty and 

) (217) 

) (218) 

proof: Recall from equations 193 through 196, we can deduce the following 

results: 

f = — (4> — 1) (219) 

«c 


, . , o c a c 

h = min( — — , - — 

Or Or 


. , «c 

k h = max( — — , - — 
o r b r 


4> 2 - 1 
$2 _ 2 ^ + 1 

$2 _ 1 
l 2 - 2$ + 1 
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( 220 ) 


_ b 2 

r 2 = -^(o 2 -20 + 1 ) 

a c 

or = — (<F - o) 

a c 

(Of) 2 _ ($2 _ l)f2 

a c a c 

= %0-l) 2 

«c 

= (f) 2 >0 


( 221 ) 


( 222 ) 


By fact l,we know that (0,T) is mean square stabilizable. Moreover, we have: 

or y($r) 2 -(^-i)H 

p p 


or 


= i 

r 2 r 2 


p 


Oc 

K 


or 

~ _ P 
£(0?-Q)± ^(0-1) 
11 ( 02-20 + 1 ) 

O 2 - 1 1 


or — 


Oe 

6, 


$ 2 - 2$ + 1 


The second statement is thus proved. □ 


Note: (1) 4> 2 - 2<f> + 1 = $ 2 - ($) 2 + ($ — l) 2 > 0. (2) If < 1 ,i-e., ^ 
is mean square stabledhen we have (ty) 2 < 'P 2 < 1 and therefore $ is also mean 
stable. (3) The nonlinear optimization problem 5.1 require one more constraint to 
have meaningful solution. 

h\ <C k l* h 
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where ki and k are given in equation 218. Therefore, 
becomes: 


min J = q(p + w) + hwk 2 
subject to the following conditions : 


p = (1 + bc 2 g~ 2 )~ l b 

b = <f> 2 p + -f F 2 
w = ( ) 2 w -j- b — p 

ki < k < kh 


minimization problem 


(223) 
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In order to solve the above optimization problem, various methods can be used. 
First of all ,we can apply the Lagrange’s multiplier method and try to solve a system 
of nonlinear equations, (see theorem 2 of chapter 4) Secondly ,we can use various 
search methods to solve it numerically. Before doing that, we will simplify the above 
optimization problem first. Because of the low dimension of the system, it is very 
easy to express J as a function of k only without any equality^ constraint. Once 
this is done, a simple but powerful search method such as the Nelder and Mead 
simplex method can be used to find the optimal gain k. In the following ,we will 
try to calculate the analytical expression for J as a function of k only. 

Since A:, <k< k h ,we have ^ 1 and (^) 2 ^ 1. Therefore from equation 223 
,we can deduce the following: (like what we did in section 5 of chapter 3) 


to = [i -m 2 }-'(b-p) 

b = pp + f 2 

(224) 

p = (1 + 6 c 2 )- 1 6 

(225) 

where we have assumed g 2 = 1 for simplicity and 

p = (i - >i' 2 )- 1 [<& 2 (i - 'J' 2 ) - ^ 2 ] 

(226) 

f 2 = (1 - Vp')~ 1 (l - ^ 2 )F2 

(227) 


Note: (1) Equations 224 and 225 are the traditional steady state Riccati equation 
for the Kalman filter ([6]). However, it is not always true that /? > 0 in our case. 
(2) Both p and f 2 are function of k only. 

Solving equations 224 and 225 ,we have: 

V = (l+/V)- 1 / 2 ;ifp = 0 

-( ] + c‘p -p)± 7(1 +oV 2 -/-) i + W 2 „ 

p = —\p r 0 ( 22b ) 

2 pc 1 
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b 


( 229 ) 


= PP+f 2 

As shown in equations 211 and 212, 'F 2 and *F 2 both are function of k. Substi- 
tuting the expressions of p and f 2 into equation 228, we have (after some lengthy 
computation): 

_ ZlLo (2.30) 

P ~ $2(l_0 2 )-l2 

Note: ( 1 ) It is easy to prove that p is continuous at p = 0. (2) We have to choose 
sign in equation 228 as the following fact shows. 

We know that there exists positive semidefinite solution if p > 0 (see [6]) under 
appropriate conditions. For p < 0, we have the following fact, 

fact 3: If p = — \p\ < 0 and c 2 ^ 0 then p > 0,6 > 0 and w > 0 if we use the 
positive sign in equation 228. 


proof: If p = — \p\ < 0,we have 

i + c\f‘ + 1 p \ =f •jv + er + w-mp]? 

* ~ 2W 2 

(1 +r 2 / 2 + W) 2 - w 

= (c 2 / 2 - W + 1) 2 + 4M> 0 

Therefore, p > 0 for both sign using the above identities. However, we have 


b = -pp + f 2 

-( 1 + c 2 f 2 + \p\ ) ± yj(l +c 2 / 2 + H) 2 -4/V 2 + 2 c 2 f 2 

2c 2 f 2 


We can prove that: 


( 1 + c 2 f 2 + \p \ ) 2 - 4/V - (c 2 / 2 - 1 - IH) 2 
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= 4/V > 0 


Therefore b > 0 for positive sign only. Following the same argument as in chapter 
4, we know w > 0 also. □ 

From equations 223 through 224 , we have: 

w = (i -* 2 r l [(p-i)p+f 2 ] 

J = qp + (q + hk 2 )(l — ^ 2 ) _1 [(p — 1 )p + f 2 ] 

= [q + {q + hk 2 )(l-y 2 )-\p-l)]p + (l -V 2 )- l f{q + hk 2 ) (231) 

Substituting the expressions for p,p and f 2 in equations 226 227 and 230, we obtain 
the analytic expression for J as a function of k only. 

_ Eto (232) 

(1 _ _ ^2) _ ^2](! 

for k{ < k < kh 

where ki and k} r are given in equations 215 and 216. (a,-, 6 n q) are functions of 
statistics of system variables 17*, Fk) (up to second moment) , system variables 
(c, g) and weighting indices of the performance criterion (q, h). Our minimization 
problem has been simplified into: 

min J(k) 

k 

ki < k < k h (233) 

Of course, we need to assume that ($, T) is mean square stabilizable for S — ( ki , kh) 
to be nonempty, c 2 is assumed to be nonzero, otherwise b = p, w = 0 and x * = 0 
w. p. 1. Since J{k) is an analytic function of k and S' = (k^kh) is closed and 
bounded, we know that there exists an optimal k * € S which minimize J{k). From 
the above argument, the following facts are given: 
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fact 4: There exists an optimal solution (p*,b%w*,k*) to the optimization 
problem 223 with the property p* > 0, b* > 0 and w* > 0 provided ($,I ) is mean 
square stabilizable and c ^ 0. 

fact 5: The total system is mean square stable provided F 2 > 0 and q > 0. 

The above is a direct application for theorem 6 of chapter 4. In the follow- 
ing, some examples will be given along with the simulation results. But , first the 
total system matrix is given below: 
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fact 6: The Kronccker product of total system matrix A Xf and A xr is equal to 


<i » 2 -$r/\ 


r 2 P 


-MTK 

Jty 2 

+JV 2 


-JAfTK 


JY 2 K 2 

-JMK 


Ar 


A rf = 


j4> 2 — ja>yi< 


-J<HYK 

+JA> 2 


JT 2 K 2 

-JA>YK 


J 2 P A' 2 

j 2 ^ 2 -j 2 Wtk -j 2 a>yk +j 2 ^ 2 

-2J 2 A^r 


Example 1: Assume the continuous time system equation is given by: 

x(t) = 20x{t) + 2u(t) + 0.5n(t) (234) 

Vk = .r(cr fc ) + 0.3^ 

The continous time system is randomly sampled by {oq-} - the sampling process. It is 
assumed that {/;,.} is an independent identically distributed stochastic process. It is 
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assumed that {t^. = &k~ (T k r-i} is uniformly distributed in the interval [Ai, A 2 ]*, where 

A x =0.05; A 2 = 0.15 


Therefore, we have: 

E[t k ] = 0.1 and Var[t k ) = 8.33 x 10” 3 
The performance index is given below: 

J= Jim (i;(0.1«fc + 1°4)} (236) 

Therefore, we have defined: 

q = 10; * = 0.1 

The equivalent discrete time system equations and associated statistics is gi\en 
below after some simple calculations (see equations 193 through 200): 

.r*-+i = + T k Uk + FkUk ( 237 ) 

v k = x k + 0M k C 238 ) 



<t> 2 = 99.0099; T 2 = 0.8264 
W = 9.0326; 4> 2 = 23.6045 
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f 2 = 0.2360; 4>f = 2.3605 
F 2 = 0.6126 

Using fact 2, we can compute S = (A'/, A'/,) as follows: 

k, = -11.8595; k h = -10 

Recall that S is the interval in which ^ is mean square stable. S is not empty as 
is guaranteed by fact 2. Our equivalent minimization problem is as follows: 

min 10(p + w) + O.lteA: 2 

p y b,u\k 

subject to the following constraints: 

p = (1 + 11.1116) _1 6 

b = 99.0099/j + 'I' 2 vc + 0.6126 
w = ^ 2 VC + b — p 

- 11.8595 < k < -10 (239) 

where we define: 

O' = 8.6836 + 0.7684 A- (240) 

W 2 = 23.6045 + 4.721 A- + 0.236A: 2 (24 1 ) 

W 1 = 99.0099 + 18.0652 A + 0.8264A 2 (242) 

As discussed before. J(k) can be expressed as a function of k only in the form of 
equation 232. Figure 4 display J(k) as a function of k in the interval S = (A;, kh)- In 
order to solve t he optimal control problem, all we have to do is to find the minimum 
of function J{k) in the interval S. In our simulation, the so-called Nelder and Mead 
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Figure 4: The performance index J(k) as a function of k 
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simplex method is adopted to search for the minimum. It turn out that the optimal 
gain k* is equal to: 

k* = -10.8856 


(243) 


We also have 

p* = 0.0893 
b * = 11.8895 
iv* = 13.1420 
J*{ k*) = 288.0407 

The dynamics of our controller is as follows: 


(244) 


x k+l = 8.6836.r* + 0.7684 u k + 0.9925(u*+i - 8.6836** - 0.7684u*) (245) 


u k = -10.8856** (246) 

We are now in a position to check the validity of theorem 2 of chapter 4. Horn 
that theorem, we know that there exist —a > 0 and —7 > 0 satisfying the following 
equations: 

k = (0.1 - Fa - P7) -I (^ra + $F 7 ) (247) 

-a = J 2 k 2 (0.1 -Fa -n 7 ) - J 2 4> 2 a (248) 

_ 7 = _ $ 2 7 _$2 a + 10 - (0.1 - f 2 a- T*~/)k 2 (249) 

where 

J = (1 — ac); a = peg ~ 2 

Therefore, we have 

k = (0.1 -0.236a -0.82647)- 1 (2.3605a + 9.03267) (250) 
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— a 


= J 2 k 2 ( 0.1 - 0.236a - 0.82647) - 75.405 J 2 a (251 ) 

= -99.00997 - 23.6045a + 10 - (0.0236a - 0.82647) A- 2 (252) 

where 

a = peg* 2 = 11.11 1 p; J = l — a 

We already know that 

a = 0.9925; and J = 1 — a = 0.0075 (253) 

Solving equations 251 252 and 253, we obtain 

-o = 0.18549 > 0 ; -7 = 31.31169 > 0 
Substituting the above answer into equation 250, we have 

k = -10.8864 

which is the same as A” as expected. □ 

Equations 210 and 2 12 give rise to the following result: 
vp = 0.3195 < 1; ^2 = 0.2872 < 1 

It is noticed that V I ; is both mean and mean square stable as desired. 1 he Kronecker 
product of total system matrix A rf . and A xe is computed and shown below. 

0.2872 0.3971 0.3971 97.8275 

0.0014 0.0038 -0.0171 0.7558 

A rf 0 A„ = 

0.0014 -0.0171 0.0038 0.7558 

0.00001 -0.0001 -0.0001 0.0058 
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The eigenvalues of C) A rf are found to be: 

0.2940 

—0.0104 

&(A xe 0 A X e) ^ 

-0.0039 

0.0208 

which clearly shows the mean square stability of the total system. Recall section 3 
of chapter 4, the evolution equations of covariances {p k , b k , w k} * s as follows. 


Pk+l = 

(1 + b k +\c 2 g 2 ) 

(254) 

bk+i = 

<t> 2 p k + $ 2 w k + F 2 

(255) 

Wfc+X = 

+ h+ 1 - Pk+ i 

(256) 


Starting from (p u = h i> = wq = 0) , figure 5 shows the evolution of covariances 
(p kj b k ,w k ) for 100 runs. It is noticed that they converge very fast to the steady 
state value (/)*, b*,u*). This is expected since the total system is mean square 
stable. Next, let's compare the performance of the optimal control to that of the 
certainty equivalent, control. If we didn't take into account the uncertainty in the 
sampling process,! hen the easiest thing we can do is to replace all the stochastic 
parameters by its mean values and then apply the traditional LQG theory to find 
the so called certainty equivalent control. The certainty equivalent gain is found to 
be: 

k c = -11.2177 

Our optimal gain is k * = —10.8856. Therefore, the optimal control is moie cautious 
facing the uncertainty of the sampling process. Table 1 is a comparison of the cost 
components between the optimal control system and certainty equivalent control 
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Figure J>: evolution of covariances (pk,bk,Wk) f° r 100 runs 
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type 

state cost 

contwl cost 

total cost 

optimal control 
c. c. control 

132.313 

146.1175 

155.728 

182.7440 

288.041 

328.8615 


Table 1: Comparison of cost between optimal and certainty equivalent control sys- 
tem in example 1 of section 1 of chapter 5 

system. As you can see, this cautiousness paid off since the performance index of 
them are : 

J(k c ) = 328.8614; J(F) = 288.0407 (257) 

which shows a 14% degradation in performance index for certainty equivalent con- 
trol. Moreover, in the calculation of equation 257 we have applied the optimal filter 
gain a = 0.9925 instead of the certainty equivalent filter gain a c which is equal to. 

a, - 0.9828 

The Monte car lo simulation is presented next. Figure 6 display the trajectories of 
the state variable and optimal state estimate :r^ . Figure 7 displays the average 
cost as a function of time. It turns out that the simulated cost is equal to 


j: 


224.1396 


which is very close to the theorectical value 288.0407. The simulated aveiage cost 
for the certainty equivalent system is (using k c = —11.2177 and a c = 0.9878). 


V 

smi 


281.8415 


Again this is a 25 '/( poor in forformance for the certainty equivalent strategy. All 
in all, the optimal system perform better than the certainty equivalent system. 
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Figure 6: simulation of trajectories of x * and &k 
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Figure 7: simulation of average cost as a function of time 
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Example 2: The continuous time system is given by: 


x(t) = hOx(t) + 2u(t) + 0.5n(t) 
v k = x* + 0.30* 

The sampling process {£,} is independent identically distributed. Furthermore,it is 
assumed to be uniformly distributed in the interval (Ai, A 2 ). 

Ax = 0.05; A 2 = 0.15 

The mean and variance of is then: 

E[U] = 0.1; Var[U] = 8.33 x 10' 3 

The performance index is given by: 

J = Jim_ j E |po.01«| + 100i ! t )| 

We then have 

S = {k h k h ); ki = —25.0549; k h = -25 
The optimal gain k* and the optimal cost J * are found to be 

A** = -25.0273 G 5; X(k*) = 8.17 x 10 6 

The certainty equivalent gain k c and its associated cost J c is 

k c = -25.0698 i 5; J c (k c ) « oo 

Therefore the certainty equivalent system is not mean square stable although it is 
mean stable. However, for the optimal system due to cautious in applying the gain 
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the total system is still mean square stable as is shown below by the spectrum of 
A xe ® A ie ( the Kronecker product of total system matrix A xe and A xe ): 


0.6067 


cr{ A xe © A, 


-0.0004 

0.0000 

0.0004 


Figure 8 plots the performance index J(k) as a function of k. In general, the optimal 
system is guaranteed to be mean square stable under certain conditions as described 
in fact 5. Whereas the certainty equivalent can only guarantee mean stable. □ 
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Figure 8: The performance index J(k) for example 2 versus k. 

88 



5.2 Two Dimensional System Simulation: An 


Example 

Because of the complexity of the problem, no general results for two dimensional 
systems are given. We do know that once there is a solution to the generalized 
Riccati equation.the total system is guaranteed to be mean square stable under 
appropriate conditions. In the following, a specific example is presented. Where 
the solution of the steady state Riccati equation is obtained through Newton s 
method. The total system is shown to be mean square stable. The performance of 
the optimal control system is compared to that of the certainty equivalent control 
system through simulation. The results are 54% better in favor of the optimal 
control system. 

Example 1: The continuous time plant is given by 

i = Ax + Bu + Fn (258) 

v k = Cx(cr k ) + G0k (259) 


where 



0.01 

-1 


0 


l 

— 

1 

0.01 

; B = 

l 

; F = 

2 


C = [ 1 , 1 ] ; G = 1 


It is assumed that {f k = cr k — < 7 k _i} is an independent identically distributed stochas- 
tic process. To be specific, we assume {t k } is uniformly distributed in the interval 
[Ai, A 2 ], where 

A x = 2.4523 ; A 2 - 3.5477 


89 



Therefore ,\ve have: 

E[U) = 3 ; Var[t t ] = 0.1 
The performance index is given by: 


1 


N 


J= lim -££[<^ + ^.1} 

N— *<x> J\ ~~ 


where 


Q 


10 0 
0 10 


H = 1 


The equivalent discrete time system equations are then given below: 


x k+ i = 4>(t k+ i)x k + T{t k+l )u k + \/FT*(h + i)’M- 
r, = Cx k + G0 k 


where 


m = e 


o.oit 


cos t — sin t 
sin t cos t 


m = 


i 


FF*(t) = 


e° 01< (cos t — 0.01 sin i) — 1 
e oolf (0.01 cos t + sin t) - 0.01 J 

— | /s (0 + 2/5(0 


1.0001 

h{t) + 4/ 8 (0 - 2fe(t) 


-I +2/5(0 /s(0 + 4 / 7(0 + 2/ 6 (t) 


in w 


hich 


0(0 — 


0(0 — 


e oo2i (0 .01 cos 2 1 + sin 20 - 0.01 

2.0002 

1 _ e ° 02t (cos2t -0.01 sin 20 


2.0002 


Mt ) = — (< 002 ‘ - 1 ) + ttLt ^- 02 '* 0 - 01 «*» + «" 2 <) - 001 1 


1 


0.04 


4.0004' 


fjf) - — (c omt - 1) - — [e OO2t (0.01 cos 2 1 + sin 20 - 0.01] 

J * K) 0.0 l l ’ 4.0004 L 


(261) 

(262) 
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The associated statistics 

with (4>fc, 

r ^FF* k ) 

are given 

below: 


0.9452 

0.1186 

0.1186 

0.1166 



-0.1186 

0.9452 

-0.1166 

0.1186 


$ © $ = 

-0.1186 

-0.1166 

0.9452 

0.1186 

; r©r = 


0.1166 

-0.1186 

-0.1186 

0.9452 



3.8897 

-0.2181 

-0.2181 

0.1119 


1.9163 

0.2570 


1.9163 

0.2570 

-0.0994 

-0.1141 


-0.2570 

1.9163 



; r©4> = 



-0.2570 

1.9163 


-0.0994 

-0.1141 

0.1141 

-0.0994 


0.1141 

-0.0994 



-0.97 -0.1373 


-1.9712 

4> = 

0.1373 -0.97 

; r = 

0.1176 


FF* 


7.7424 -0.3657 

-0.3657 7.7221 




0.0042 

-0.0145 

-0.0145 

0.0978 

0.0145 

0.0042 

-0.0978 - 

-0.0145 

0.0145 

-0.0978 

0.0042 - 

-0.0145 

0.0978 

0.0145 

0.0145 

0.0042 


f © f = rwr - r © r 

0.0040 

0.0136 

0.0136 

0.0981 
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F©T=?¥T-$©r 

0.0041 -0.0136 
0.0146 -0.0979 
0.0136 0.0041 

0.0979 0.0146 

where ” ® " is the Kronecker product of matrices. 


Note: (1) (A,F) are controllable =*- F F* > 0 (refer to notes for theorem 1 
of section3 in chapter 4). Assuming the existence of the optimal control , FF > 0 
together with Q > 0 guarantee the mean square stability of the total system as will 
be shown later. 

(2) It is easy to show that ($,T) is mean square stabilizable and that ($>£•) 
is mean square observable. This may have something to do with the existence of 
optimal solution. 

To find the optimal solution, we will applying theorem 2 of chapter 4 and tiy 
to find the solution of the coupled steady state Riccati equation. By using the 
Newton’s method, we are able to obtain the following solutions: 


P = 


251.9681 -251.9698 

-251.9698 252.9645 


B = 


251.9685 -252.2166 

-252.2166 393.7223 


W = 


611.7 442.1 
442.1 487.3 


-a = 


1310.1 1.983 

1.983 0.0081 
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21.487 -25.16 

-25.16 740.4 


IC = [H - f *af - + r*7$] 


= -0.7625 , 0.7156] 


The certainty equivalent control is found to be: 


I< c = [-0.5129 , 0.3210] 


The optimal cost is found to be: 

J(/\“) = 1.6162 x 10 4 


However, the cost associated with K c is equal to : 

J C (K C ) = 2.4851 x 10 4 


This turns out to be 54% increase in cost which should be of no surprise. Next 
, let’s verify the mean square stability of the total system. The total system matrix 
is given below (see equation 153 of chapter 4): 


A 


xe 


-r k K 

J^k — JT&A + Jty 


The following fact is well known and is given without proof: (see Kalman s paper) 


fact: is mean square stable if and only if p{ A xt 0 A xe ) < 1, where p{.) is the 

spectral radius of matrix. 

It is easy to prove the following lemma and hence is given without proof: 
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lemma: A xe 0 A xe is similar (not necessarily equal to) to the following matrix: 


\P 0 $ -V®TK 


-TK ® t r I< 0 TK 


=- 0 JTK =- 

$ 0 Jvp -TK © JvP 

d-'P 0 J’P 


rA' @ JTK 
-TK 0 JV 


Ji > 0 \P -J<P 0 TA' 


- JTK © <P JrA © TK 
+Jty®ty -JVQTK 


( 263 ) 


JTK 0 JTA' 

-jrA'0J'P 

jv © jv -j'a © jrA -jrA' © ./$ 

-Jtf 0 Jf A' 

+j»p © j# 

After some lengthy computation and by applying the above lemma, the eigenvalues 
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of A xe ® A xe are found to be: 


^(^xe © A xe ) 


0.9823 


0.5581 + 

0 . 220 ?: 

0.5581 - 

0 . 220 ? 

- 0.4476 


- 0.4179 


0.2395 


0.9037 


- 0.4749 


- 0.3565 


0.0046 + 

0 . 0034 ? 

0.0046 - 

0 . 0034 ? 

0.0001 


- 0.0028 


0.0059 + 

0 . 0031 ?' 

0.0059 - 

0 . 0031 ?: 

- 0.0039 



As shown above, all the eigenvalues lie inside the unit circle. Indeed, the total 
system is mean square stable as expected. □ 
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Chapter 6 


conclusion 


Chapter 3 and 4 are devoted to solving the optimal LQR problem with i. i. d. sam- 
pling. The fixed configuration approach is adopted to obtain the suboptimal con- 
trol which has the nice property of cautious control. The optimal linear least mean 
square estimator for system with random parameters is derived in chapter 3. The 
evolution equations of covariance matrices is derived where it is noticed that the 
covariance matrices are dependent on the particular control sequence. The neces- 
sary conditions for iV-horizon optimal control problem is derived using Lagrange s 
multiplier method in chapter 4. The suboptimal solution is noticed to be the opti- 
mal solution provided the sampling is deterministic. Finally , the infinite horizon 
control problem is formulated as a classical minimization problem where assuming 
existence of solution the total system is shown to be mean square stable provided 
certain mean square observability conditions hold. Various theorem are given to 
test the mean square stability of the total system. Furthermore, it is shown that the 
total system is stable with probability one. 
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The simulation results presented in chapter 5 show the applicability of our ap- 
proch. For one dimensional system, the existence problem of optimal control is 
solved. It is shown that the performance of the optimal system is much better as 
compared to that of the so called certainty equivalent control systems. The optimal 
system can guarantee the mean square stability due to the cautiousness facing the 
uncertainties of the sampling process, whereas the certainty equivalent system can 
only guarantee mean stability and may not be stable under certain conditions. 

The techniques used in deriving the optimal control is not limited to systems 
with randomly sampled observations. It can be applied to derive the optimal control 
for all systems with random parameters as well. Furthermore, our theorey naturally 
provide a method to obtain the robust control-robust to parameter variations. For 
example, we can treat certain unknown parameters as random variables say with 
a uniform distribution in some reasonable interval and then apply our theorey 
to derive the optimal control. In so doing ,the total system is stable even with 
parameter variations. 

The optimal control problem for systems with random parameters and imper- 
fect observations is in general a very difficult problem. Because of the curse of 
dimensionality of sufficient statistics, no analytic solution can be found in general. 
Efficient suboptimal control is yet to be found. Our techniques do provide a begin- 
ning toward this direction. 
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Appendix A 

Derivation of optimal control 
control under variance neutrality 
condition 


The optimal control problem of concern is the following: 

Discrete Plant 

Xk+\ = ${tk+i) x k + r(4+i) M fc + F{t k+ i)i]k (“hf) 

v k = Cx k + G0k ( 265 > 

where {?^}, {@k} and are mutually independent random sequences. Fur- 
thermore, it is assumed that {t^} is independent identically distributed. 

Performance Index 

J = E | i*kQ x k + u l - 1 Hu k~ i ) } ( 266 ) 
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Admissible Control 


Uk is measurable v k — {tq, tq, • • • , Vk} 

Goal find optimal admissible sequence {«fc}o _1 such that performance index J is 
minimized subject to dynamic constraint 264 and 265. 

In the following ,we will use dynamic programming techniques to derive the optimal 
control. Consider the last stage problem: 

\ N = E[x* n Qxm + u* N _ l Hu N -i\\v N - 1 } 

= .E [($(<#)•£ jv-i + r(<,v)ujv-i + E(fjv)/;Ar_i) Qx 

(4>(fjv)- r .v - 1 + r(/. v )?qv-i + E(f i v)?/iv-i)||C v_1 ] + U/v_i#UjV-i 

+2.r^_, QT(t N )u N ^\\v ] 
+E[i]* N _- l F(t N yQF(tx)i] N _ l ||v A_l ] + u* n _ 1 Hu n _ j 
Because of independence assumption, we have: 

A at = EIx^W^xn.x + u^T^QTuK.i + 2u N ^WQTx N .- l \\v N - 1 ] 

i +tr{QFF*} 

By taking the gradient of A\- with respect to u^_ i,we obtain: 

4 ^- = ^MJV-I f^v-1 + r^E^.xll^- 1 ] = 0 

9un-i 

==» «JV-I = (// + 

def j r a 

= —1\N-\VN-\ 
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where we have defined: 


A'/v-i = (H + r*QT)- l T*Q$ 

where we have assumed that the inverse exist, otherwise the Moore- Penrose pseudo 
inverse has to be used. Next, we will compute the optimal cost associated with the 
optimal control derived above. 

def ■ \ 

r?v = nun A/v 

u N—i 

= E[x* N _ x ^ m Q^x N _i + 

+£K_ 1 (F$Tu iV _ 1 + + r*g<&^v-i)||w A ' _1 ] 

+tr{QPP*} 

= - Zn-iK* N -iFQ*Zn-i 

+ tr{QTF *} (267) 

From orthogonality principle, we have: 

E[(xn - i — x J v-i)*A J v_ir* , e^(-rN-i — ^-i)||C N *] 

= a^Fqsp/ 

= £[(.r/v_i - x,v_i ||i> A - 1 ] 

= E[x m N _ 1 K%_ l T*G$x N -i\\v N - 1 ] - Pv-i^TA,v-i.r,v-i 

where we have defined: 

P[ = £[(xjv_i - 
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From variance neutrality assumption, we know that P{ is independent of v h 1 and 
ujv-i- Therefore, from equation 267 we have: 

+tr {QTF*} + tr {K' n _ x WQ$P( 

= E[x m N _ l P^x N _ 1 \[v N ~ 1 ] + iq 

where we have defined: 

P{ = ~¥ r Q$ - 

t/ x = f tr {QTT-} + tr {K' n _ x ^Q$P( 

Now, consider the last two stage: 

f’V-i = min E[x‘ N Qxx_i + u1_ 2 Hun - 2 + x* w Qx,\ + Wtf_i#«Ar_i ||e ” 2 ] 

“iV-2,«N-l 

= min E{x* w _,Qxm„i + u* N _ 2 Hu^_ 2 + min E[x* N Qx^ + ||n A_1 ]||u ;V_2 } 

^iV-2 J " 

= min E[x* n _ x Qx n_i + u'x_ 2 Hun ~2 + ^v-iA^v-i + ^ill^ -2 ] 

2 

= min + P^)x N _i + u‘ N _ 2 Hu N _ 2 + i/x||y N ~ 2 ] 

^N-2 

The above is the same one stage problem as before. By applying the same proce- 
dures as before ,we obtain: 


UN-2 = —Kn-2*N-2 

r V -i = E\x* n _ 2 P 2 x N - 2 + v-i ||t> jV_2 ] 

i<n- 2 = [f^rnw + ur'TWTT^ 

p c 2 = wW-a';J 

V2 = tr(Q + P{)TF + trK- N _ 2 V(Q + Pf)<l>P 2 / + 
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In general, by induction we obtain the following results: 


V 0 < i < N - 1 


Ui = — K{X i 

r i+ i = ElxiPff-iXi + i/pj-iWv'] 

I<i = [V(Q + w + H ]- 1 r*(g + Pk-i-i)^ 

PN-i = **«? + PhZ i)* - k;v{q + 

»N-i = HQ + Ph-i-xWF + trK!T-{Q + P^mW P N- t + "N-i-i 


□ . 
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